Computer systems typically include one or more computers, referred to as hosts. Where multiple computers are used, the computers are interconnected by a network that allows them to share data. Typically, such networks also include one or more data storage devices to provide additional data storage capacity for the networked computers. A common data storage device is a disk array, also sometimes referred to as a Redundant Array of Independent (or Inexpensive) Disks (RAID). A disk array is two or more hard drives or similar disks that provide data storage for connected hosts.
Computer systems operate optimally when all components in the system strike an equal balance of performance. Host systems and the attached storage should normally be balanced so that the ability of the storage to satisfy host requests is roughly equivalent to the workload generated by the host. In more complex configurations, many hosts can be transferring data to and from one or more storage devices. When more than one host is accessing a single storage device, it is important that the storage device be able to accommodate the performance demands of all the attached hosts. If the storage device is significantly less capable than the hosts from a performance perspective, the storage can be a limiting factor in overall system performance.
Fibre channel (FC) technology is a technology for connecting or networking host devices and storage devices. Fibre channel technology allows configurations that can include thousands of hosts connected to a single storage device. Fiber channel uses the Small Computer Systems Interface (SCSI) protocol, e.g., the SCSI-3 communication protocol. Consequently, each of the thousands of potential hosts can address up to 65,536 logical storage units (LUNs) within a disk array. Disk arrays that communicate using Fibre channel technology are shipping today with the ability to create thousands of LUNs.
Disk array configurations vary from arrays that contain four or fewer drives to arrays that contain more than a hundred drives. Ultimately, the performance of a disk array is limited by the number of disks in the array. This is the case because all work is eventually passed through to the disks.
Clearly, Fibre channel technology invites users to attach large numbers of hosts to a single storage device. Large numbers of hosts can direct huge amounts of work toward a storage device, such as a disk array. Consequently, users in such an environment may be accustomed to working with enterprise disk arrays that include hundreds of disks and are, therefore, capable of the performance required to support large numbers of hosts.
In a high demand environment, each host may have multiple input/output (I/O) requests to multiple LUNs at any given time. Each such request will have to be queued until processed—or rejected because the queue is full.
System performance is a complex topic that has been studied extensively. Given the number of performance factors and permutations, it can be very difficult, if not impossible, for users to know at the point of design if the workload that will be generated by their host/application configuration will exceed the ability of their storage systems. Therefore, it is not uncommon for users to set up systems that are not balanced with storage performance.
In one unbalanced system, the user might have storage performance capability that significantly exceeds the workload generated by the attached hosts. The user might be dissatisfied with the performance of the whole solution and may need to add more host capacity to achieve the desired performance result.
The present invention addresses the complementary scenario where the host/application workload exceeds the ability of the attached storage. As used herein, the term “over-configuration” is used to describe the scenario where host/application performance needs exceed the storage performance capabilities. Another way of stating this is that the system designer has under-provisioned the storage for the needs of the overall solution.
Over-configured systems can lead to more serious problems than dissatisfaction. In the case where the data storage array is not capable of meeting system performance requirements, the response times of user applications will become unacceptably high. In the more extreme cases, the symptoms can include catastrophic events like application failure, server crashes and host cluster failures. These catastrophic events occur because the software in the host (drivers, applications, volume managers) limit the amount of time they will wait for storage devices to complete requests. When the response time of the storage device, e.g., a disk array, exceeds the limit enforced by the host software, catastrophic events can occur. Clearly, there is a need for ways to detect over-configuration and to avoid over-configuration entirely.
Presently, users discover over-configuration problems through trial and error. The discovery often results from a support request where support personnel are only able to guess that a system is operating in an unbalanced state. This is a reactive solution that focuses on telling users how to avoid these unbalanced configurations rather than how to detect and mitigate performance problems as they arise.
Because of the complexity and dynamic nature of performance, this can be a very error prone process. Unbalanced configurations are very common. Once the unbalanced configuration is setup, users must first establish that performance of the whole system is unacceptable. Users identify a poorly performing system through a variety of tools that provide clues. However, the process always includes a manual examination of the configuration, the applications and many other pieces of information obtained from a variety of sources. The process inevitably ends with a heuristic analysis by an expert who concludes that the system is unbalanced. Given this conclusion, the next step is to add more storage and hope that the performance problems are resolved.
In general, when an I/O request is sent to a storage device it is either accepted or rejected. Requests that are accepted are placed in a service queue to await further processing. When the ability of the storage device to capture and queue new requests is exceeded, the request is rejected and the request is returned to the host system with a status of queue full (QFULL). Though QFULL is the appropriate return status for this condition, some storage devices return BUSY rather than QFULL. This is important in that host system drivers will typically display different retry behaviors for BUSY and QFULL conditions.
A storage device could make its service queue so large that QFULL events could not occur, but this solution also has limitations. Service queue size is limited in a practical sense by the timeout periods enforced by host system drivers and applications. In order to ensure that storage devices do not cause applications to wait forever, host system drivers and some software applications enforce timeouts on I/O requests. If a request times out, the host assumes the storage device is having some problem. To recover from this condition, the host system instructs the storage to abort the I/O and then retries the request (up to some finite number of retries).
If the number of I/O requests queued is large enough, the latency in processing the requests can exceed the timeout limit imposed by the host system drivers and/or host system applications. These timeouts occur simply because the storage device cannot process the number of requests in the service queue in the allowed time. For this reason, there is a practical limitation on the size of storage device service queues.
Host reactions to both QFULL events and timeouts are expensive in terms of performance. In fact, both events can propagate through the system and result in real or perceived failures. Both events are treated as error cases in host code (i.e., in fibre channel drivers). Often these events show up in host system logs where they may be treated as critical failures. This interpretation may or may not be accurate depending on other behavior.
The handling of QFULL and timeout events leaves the storage device designer with a dilemma. On the one hand, it is desirable to maximize the size of the storage service queue. A larger queue means more hosts can be connected. This improves the connectivity of the array without resulting in unnecessary QFULL events. If the I/O request characteristics of the hosts and applications are appropriate, it might be possible to construct a balanced system with a large number of hosts, even to a relatively slow storage device. On the other hand, the storage device designer wants to keep the queue size small enough that even large configurations with the worst-case access patterns can be accommodated without timeout events occurring. Obviously, it has been extremely difficult in the past for a system designer to try to balance these competing considerations.